P0: repair FAT relay-bench runtime latency by masarray · Pull Request #265 · masarray/arsas

masarray · 2026-09-06T03:52:28Z

FAT relay-bench recovery — second software gate complete / physical re-bench required

Golden reference remains Build #1888 / ARSAS commit 712e2c557d5da83d2f02b81f3a58905e985d1e50 with immutable ARIEC61850 pin 11ab2304482600c19ba979f4fc9021ddb46b9af9. This PR remains a relay-bench candidate only and must not be merged until the physical gate below passes.

The previous physical candidate 849be91469026e804feee80d7867ebc9dbae83ec failed relay-bench acceptance: Start/Continue, post-command UI, Value 2 capture and Stop could freeze for tens of seconds/minutes, and a delayed FAT evidence projection could leave LIVE VALUE stale. The relay diagnostic showed the native command path itself completed in about 0.4 s, so the recovery below targets the FAT/WPF/evidence hot path rather than ARIEC control timeouts.

Current candidate head: 0fa8cebd9a58fab91adc378d49773a823dfef488.

Second-round relay-bench fixes

Single process-image authority: while FAT is open, raw FAT evidence/runtime observers are detached. FAT LIVE VALUE and evidence consume the already-coalesced Engineering device.Points image after Engineering's UI flush.
No delayed evidence overwrite of LIVE VALUE: rolling evidence evaluation no longer writes CurrentValue, quality, source or IED timestamp back into the operator-facing FAT row. Those fields are owned only by the shared Engineering process image.
No sequence-only evidence churn: a BRCB/URCB sequence change with semantically identical process value + quality updates the cursor only; it does not enqueue another FAT evidence job.
No 58-row burst immediately after Start: the shared-process cursor is continuously primed while the FAT evidence session is inactive. Session.Start already captures the baseline, so the first UI tick after Start no longer replays the entire current process image as fresh evidence.
Fast Start/Continue when Engineering is already live: Start/Continue/Retest is intercepted before the legacy lifecycle and attaches evidence directly to proven LivePointReady rows. It does not repeat IED preparation, discovery, report selection or monitoring restart.
WPF notification amplification removed: IoTestMultiSessionCoordinator forwards only the child property that actually changed instead of expanding every leaf notification into the full coordinator projection.
Legacy FAT-window notification fan-out coalesced: compatibility window-derived labels refresh at most once per Dispatcher turn and below input/render priority. Direct Session.* bindings remain targeted and live.
Command authority: FAT position Confirm uses the same Engineering command backend. After command completion, command-panel value is reconciled from the shared Engineering process image; command-service evidence cannot outrank the real process status.
Stop remains deferred: evidence queue drain, durable disk barrier, full hash-chain verification and final persistence stay off the WPF Dispatcher; only the pressed button is locally muted.
Rolling Value 1 / Value 2: report-backed changes advance the pair immediately. A complete pair stays visibly COMPLETE while a newer polled analog candidate settles; once stable it advances normally.
THD/composite correctness: composite values are compared as complete values, not by the first numeric token, so a phase-C change such as A=11,B=12,C=0 → A=11,B=12,C=13 advances Value 2.
FAT-only tooltip suppression: FAT workspace tooltips are disabled; Engineering-mode tooltips remain untouched.
Report P1: Preview/PDF use the shared A4-landscape layout and the final FAT sign-off page contains TESTED BY, WITNESSED BY, APPROVED BY with Name / Company / Signature / Date.
ARIEC61850 source remains unchanged at the immutable pin above.

Exact-head CI gate — GREEN on `0fa8cebd`

✅ Build ARSAS #2031 — solution build, full regression suite, portable publish and portable smoke test.
✅ Validate IO List Testing #904.
✅ Validate SV evidence bundles #1129.
✅ Full regression evidence: 765 / 765 passed, 0 failed.

Portable candidate:

Artifact: ARSAS-win-x64-portable-single-exe (9998864839)
Artifact ZIP SHA-256: b219dec279f9f7e04d07e0fe6d2bb57b99c22b1eca1a6a27beabc5a1631c821e
Executable: ARSAS-1.6.33-win-x64-portable.exe
EXE SHA-256: a1a70cde724ee5581561a3d49959de6f23dc6ed7efdbff8155587f7f35f1fdb7

Physical relay re-bench gate

Start FAT from an already monitoring Static DataSet session. Button paint/state change must be immediate; no 60 s freeze and no second freeze after IED session active.
Change one digital/analog/composite process value. LIVE VALUE must follow Engineering immediately and Value 2 must update without a 20 s UI stall.
Repeat CSWI.Pos Open/Close several times. Native command timing should remain sub-second; FAT LIVE VALUE and Command Panel must end on the same report-backed physical state, with no delayed evidence overwrite or phantom opposite state.
Verify composite THD latest pair, including phase-C-only change.
Stop FAT. Window must remain repaintable/scrollable while durable seal + verification finish off Dispatcher; no minute-long Not Responding.
Repeat from restored progress / Continue FAT.
Close/return while connected and confirm no heavy Not Responding teardown.

Keep this PR draft. Do not merge until this exact 0fa8cebd build passes the physical relay re-bench.

masarray · 2026-09-06T04:02:14Z

Relay-bench P0 Gate 1 is green at db81148d0f429f6df27c0eea577b6ea8e07164f6.

Passed:

Build ARSAS #1929
Validate IO List Testing #802
Validate SV evidence bundles #1027

Gate 1 repairs report-backed analog Value 1/Value 2 latency and removes per-event physical disk barriers while keeping final journal durability/seal verification. Proceeding to the next isolated gate: fast FAT attach to an already-live shared Engineering Static DataSet/RCB acquisition session. PR remains draft and must not merge before physical relay acceptance.

masarray · 2026-09-06T05:38:49Z

P0 LIVE VALUE + Command Panel gate completed on c1063c79b9fbc47341542f70695df8be50cd4ccc.

LIVE VALUE now prefers the shared Engineering device.Points process image/report-backed state; non-forced MMS inspection remains only for missing values.
FAT Command Panel retains row instances for normal value/result/busy updates. LIVE VALUE/result/model/enablement are bound in-place; structural synchronization happens only when the command collection changes, and confirmation/semantic transitions replace only the action cell.
Regression coverage now explicitly rejects RebuildFatCommandRows and forced FAT RefreshControlValuesAsync(..., force: true).
CI corrections during the gate were limited to completing required IoTestPointPlan fixture fields and updating the old source-text assertion to the new null-safe/shared-collection implementation.

Full CI green:

Build ARSAS #1945 ✅ (full regressions + portable publish/smoke)
Validate IO List Testing #818 ✅
Validate SV evidence bundles #1043 ✅

ARIEC61850 source was not modified; immutable pin remains 11ab2304482600c19ba979f4fc9021ddb46b9af9.

PR intentionally remains draft. Physical relay-bench Start/capture/Recapture/command/LIVE VALUE acceptance is still required before merge.

masarray · 2026-09-06T06:14:15Z

P0 locked-scope verification — LIVE VALUE + stable Command Panel

Re-verified the current PR head c1063c79b9fbc47341542f70695df8be50cd4ccc against the locked P0 scope.

LIVE VALUE is genuinely live, not attach-time-only: the FAT bridge seeds values from the shared Engineering device.Points process image, rebuilds the control-feedback index, and the existing runtime point-update path projects matching report-backed point updates into SignalDefinition.ControlCurrentValue through _controlFeedbackIndex.
No normal full-row rebuild: Command Panel retains row view instances; ControlCurrentValue, ControlLastResult, busy/enabled state, CDC and control-model text are binding-driven. Structural collection changes synchronize rows; confirmation/semantic transitions replace only the action cell.
No forced FAT MMS refresh regression: fallback refresh remains force: false; shared process-image projection is reapplied afterward.
ARIEC61850 remains immutable for this P0: pinned engine revision 11ab2304482600c19ba979f4fc9021ddb46b9af9; no engine-source change is needed.

Full CI on this exact head is green:

Build ARSAS #1945 — success
Validate IO List Testing #818 — success
Validate SV evidence bundles #1043 — success

No additional code churn is warranted for this locked P0. PR should remain draft until physical-relay acceptance; P1 remains out of scope until that gate is stable.

masarray · 2026-09-06T08:46:34Z

Repair status update — current head `478909f`

Implemented on this branch:

✅ Rolling Value 1 / Value 2 — a completed evidence pair now advances on each new real process condition; repeated identical values do not churn evidence. Example 0 → 1000 → 0 ends at Value1=1000, Value2=0.
✅ Start FAT / preparation UI hot-loop removal — the preparation progress layer no longer traverses the full WPF visual tree every 50 ms at Render priority. It now runs at Background priority / 100 ms, returns immediately while idle, and updates cached progress bars only while preparation is active. A regression test locks this behavior.
✅ COMTRADE FAT report cleanup — removed the Download column plus OPTIONAL / not a FAT gate wording. FileDirectory evidence now presents the remote file identity, relay-modified timestamp, and evidence source directly. Regression assertions now reject the removed wording.

Previously locked on the same PR:

✅ already-live Engineering fast attach for FAT Start
✅ event-driven report-backed first/changed value capture
✅ no per-event physical disk barrier in evidence journal
✅ command-panel routine updates without full-row rebuild
✅ LIVE VALUE projected from the shared Engineering process image, with no forced FAT MMS refresh

Still open / not yet claimed fixed:

⏳ close-path latency while active evidence journals are synchronously sealed/verified
⏳ command Open/Close latency on a physical relay
⏳ multi-RCB selection/export workflow
⏳ normalized already-downloaded COMTRADE re-download/overwrite selection
⏳ one-row FAT header + report sign-off block

Current full CI has been triggered for head 478909f; Build ARSAS, Validate IO List Testing, and Validate SV evidence bundles are still running at the time of this update. ARIEC61850 remains unchanged at immutable pin 11ab2304482600c19ba979f4fc9021ddb46b9af9.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 849be91469

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-06T22:31:50Z

+                // Flush only on the background writer. This makes newly written evidence
+                // visible to readers without ever stalling the WPF/report callback thread.
+                if (wroteAny)
+                    _writer.Flush();


Preserve durability before accepting evidence

When ARSAS or the host loses power before Stop/Close, this background loop only calls StreamWriter.Flush(), which makes records visible but does not preserve the prior FileStream.Flush(flushToDisk: true) durability guarantee. Because Append now returns as soon as it queues an envelope and the controller immediately promotes that evidence, an arbitrarily long running FAT session can present accepted evidence whose journal records are still only in the OS cache and may disappear after a crash; retain a background durable barrier before records are treated as accepted, even if barriers are batched.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-06T22:31:50Z

+            // Structural snapshot only. Queue drain + durable flush + full read-back are
+            // still running and the lifecycle caller must await AwaitDeferredSealsAsync().
+            return deferred.Provisional;


Do not report a pending seal as verified

During the new deferred Stop/StopAll paths, IoTestSessionController.Stop() disposes the journal and immediately calls VerifySealedJournal(), so this pending branch returns an IsValid=true provisional result and the controller reports the session as verified/stopped before any queue drain, durable flush, or read-back has succeeded. If the later seal fails, AwaitDeferredSealsAsync() only causes the window handler to show an error; the controller remains stopped with JournalIntegrityText set to “Verified” and no journal attached for retry. A pending seal must not be returned as successful verification, or the controller must await it before committing its terminal state.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-06T22:31:51Z

+            // Advance the current projection before the controller promotes the newest V2.
+            // The shifted item is the already-journaled previous V2, so no process evidence
+            // is invented; this keeps current-pair assessment atomic for the live UI.
+            if (shiftedValue1 != null)
+                point.Runtime.SetFatValueEvidence(shiftedValue1);


Shift rolling evidence only after journal acceptance

When a completed automatic pair rolls forward, this mutates the real runtime Value 1 inside Observe, before IoTestSessionController.ApplyAutoCapture() attempts to append the new Value 2 event. If that append fails—for example after a background writer or disk failure—the method returns without promoting Value 2, but Value 1 has already changed from the old pair to the prior Value 2, leaving project state altered without the corresponding accepted journal operation. Return the shifted evidence in the decision without mutating the point, then apply both pointer changes only after the append succeeds.

Useful? React with 👍 / 👎.

…wice

masarray · 2026-09-06T23:31:08Z

Report-polish checkpoint on exact head 36b4c8c7160f8d5153f1124dba6be13e21eab6a3 after positive physical IED re-bench:

FAT report Value 1 / Value 2 canonicalizes exact boolean values to True / False for consistent presentation without altering stored evidence.
Removed customer-facing N included · N removed wording from report pages; document badge now reads FAT REPORT / FOR FAT RECORD.
Footer now reads Automatically generated by ARSAS Protocol Tester with generation timestamp and compact source SHA retained for traceability.
WPF Report Preview already prefers Inter (Inter, Segoe UI, Aptos, Arial) and this is now regression-locked. Native PDF remains on embedded-safe/base PDF fonts for now; it is not falsely labeled Inter without an embedded Inter font program.

Exact-head CI is GREEN: Build ARSAS #2033, Validate IO List Testing #906, Validate SV evidence bundles #1131. Portable artifact: ARSAS-win-x64-portable-single-exe #9999261153, ZIP SHA-256 b9d025c1a11d5733d6be78026ba7e7da6a1271bd47e690148dcdb7fbea971ad0.

PR remains draft until the final report visual is accepted; ARIEC61850 pin remains unchanged at 11ab2304482600c19ba979f4fc9021ddb46b9af9.

masarray added 6 commits September 6, 2026 10:50

Fix report-backed FAT analog evidence latency

bdea756

Remove per-event FAT evidence disk barriers

4007aeb

Guard relay-bench FAT immediate evidence behavior

318771d

Keep final FAT evidence durability without per-event fsync

74289c3

Preserve FAT capture status contract for immediate report evidence

8e2429f

Align FAT V2 regressions with immediate report evidence

db81148

masarray added 6 commits September 6, 2026 11:06

Fast-path already-live Engineering FAT signal matching

ad2c4b1

Guard already-live Engineering FAT fast attach

a2592af

P0: project FAT command live values from shared process image

992de77

P0: keep FAT command rows stable during live updates

c39ec8c

test: complete relay-bench fast-path fixture

2fd1426

test: lock stable FAT command live projection

c1063c7

masarray added 7 commits September 6, 2026 14:36

fix(fat): keep Value1/Value2 rolling with latest process change

90c62d7

fix(fat): advance completed evidence pair on every new condition

cb79103

test(fat): lock rolling Value1/Value2 regression

6716fce

fix(fat): remove preparation progress visual-tree hot loop

0f20e27

test(fat): lock preparation progress off hot visual loop

f3a122d

fix(fat-report): remove optional download gate wording

c56a945

test(fat-report): lock concise COMTRADE evidence layout

478909f

masarray added 7 commits September 6, 2026 15:54

Fix FAT rolling capture operator override regression

d89c61c

Align FAT progress regression with P0 responsiveness guard

ef168e6

Defer FAT close journal sealing off UI thread

38cf760

Keep FAT close responsive while sealing journals

ff77630

Guard responsive FAT close journal sealing

f89a2c5

Fix FAT P0 first-frame canonical templates

991ef5b

Lock P0 first-frame and SCL connect-only regressions

3352a7f

masarray added 3 commits September 7, 2026 04:51

test(fat): promote complete rolling pair journal-first

4eaa63b

fix(fat): advance rolling pair before current-pair assessment

618daca

test(fat): align stale recapture review with current-pair semantics

849be91

masarray marked this pull request as ready for review September 6, 2026 22:28

chatgpt-codex-connector Bot reviewed Sep 6, 2026

View reviewed changes

masarray added 8 commits September 7, 2026 05:36

fix(fat): reassert shared process image after control

da16eac

fix(fat): bypass legacy blocking relay-bench hot paths

32aefab

perf(fat): stop amplifying every evidence notification

1d24bfc

fix(fat): make report preview a usable FAT record

a7ea91b

fix(fat): issue report as FAT record with sign-off

7037515

test(fat): lock physical bench non-blocking hot paths

9a00f9d

fix(fat): keep current Value1 Value2 rolling after recapture

86fe907

test(fat): keep latest process pair after operator recapture

267646f

masarray marked this pull request as draft September 6, 2026 22:42

masarray added 15 commits September 7, 2026 05:46

fix(fat): coalesce legacy window notification fanout

b205c7c

perf(fat): consume Engineering process image instead of raw runtime t…

a9793b7

…wice

fix(fat): keep LIVE VALUE owned by shared process image

5f4e64e

perf(fat): route evidence from shared Engineering process image

58fadff

test(fat): lock shared Engineering process image authority

6d1286a

test(fat): lock freeze and live-value authority regressions

4ebc86d

test(fat): complete operator rolling point fixture

69e192c

fix(fat): do not enqueue evidence for sequence-only churn

468e370

fix(fat): forward only changed session projection properties

18f9904

fix(fat): keep shared-live path free of legacy preparation call

801073e

fix(fat): keep completed pair stable while polled candidate settles

bbe929f

fix(fat): prime shared image before evidence session activation

82b23dd

test(fat): prevent post-start full-image evidence burst

0fa8ceb

polish(fat-report): simplify customer-facing report text

fda7587

test(fat-report): lock customer-facing report polish

36b4c8c

Navigation Menu

P0: repair FAT relay-bench runtime latency - #265

P0: repair FAT relay-bench runtime latency#265
masarray wants to merge 95 commits into
fix/p2-fat-compact-headerfrom
fix/fat-relay-bench-p0-runtime

masarray commented Sep 6, 2026 •

edited

Loading

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

masarray commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

FAT relay-bench recovery — second software gate complete / physical re-bench required

Second-round relay-bench fixes

Exact-head CI gate — GREEN on 0fa8cebd

Physical relay re-bench gate

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

masarray commented Sep 6, 2026

P0 locked-scope verification — LIVE VALUE + stable Command Panel

Uh oh!

masarray commented Sep 6, 2026

Repair status update — current head 478909f

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

masarray commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

masarray commented Sep 6, 2026 •

edited

Loading

Exact-head CI gate — GREEN on `0fa8cebd`

Repair status update — current head `478909f`